Image processing device, image processing method, and non-transitory computer-readable recording medium storing image processing program

ABSTRACT

An image processing device includes: an image input unit that inputs a facial image from a predetermined device; an image analyzing unit that calculates one of facial shape, proportion lines that are lines drawn on the face to analyze the balance of the face, and blocking lines that divide the face into multiple regions following the structure of the face according to lightness and darkness of shadows due to light striking the face, based on facial feature points extracted from the facial image; and an image generating unit that decides a superimposing region of a makeup parts image based on one of the facial shape, the proportion lines, and the blocking lines, and generates a simulation image where the makeup parts image has been superimposed on the superimposing region.

BACKGROUND 1. Technical Field

The present disclosure relates to an image processing device, an imageprocessing method, and a non-transitory computer-readable recordingmedium storing an image processing program, that generate simulationimages for makeup.

2. Description of the Related Art

In beautician schools, in classes for facial makeup (hereinafter, simply“makeup”), instructors commonly explain makeup methods using photographsand illustrations (showing specific examples of makeup) in paper-mediumtextbooks. Optimal makeup methods vary in accordance with facialfeatures, so textbooks preferably include a greater number of specificexamples of makeup. However, there is a limit to how far specificexamples of makeup can be increased in textbooks, considering ease ofcarrying the textbooks. Accordingly, students cannot obtain a deeperunderstanding of specific examples of makeup suitable for various facialfeatures. It is also troublesome and time-consuming for instructors toexplain specific examples of makeup suitable for various facialfeatures. Moreover, it is difficult for students to picture how makeupdescribed in a textbook would look on a face that is different from theface in the textbook.

Accordingly, it is conceivable to use technology described in, forexample, Japanese Unexamined Patent Application Publication No.2016-81075, in order to describe a greater number of specific examplesof makeup in accordance with various facial features. The technology inJapanese Unexamined Patent Application Publication No. 2016-81075 is atechnology that changes features extracted from a facial image andchanges the impression of the appearance of the facial image.

SUMMARY

However, even if the technology in Japanese Unexamined PatentApplication Publication No. 2016-81075 is used, the instructor still hasto explain in detail makeup methods suitable for the facial image afterchanging, verbally or the like. Accordingly, the problem of explanationbeing troublesome and time-consuming for the instructor is not resolved.Also, this results in the instructor explaining makeup methods verballyor the like, an accordingly is not sufficient regarding the point of thestudents obtaining a deeper understanding, either.

One non-limiting and exemplary embodiment provides an image processingdevice, an image processing method, and a non-transitorycomputer-readable recording medium storing an image processing program,whereby trouble and time for the instructor to explain makeup methodscan be reduced, and students can obtain a deeper understanding of makeupmethods.

In one general aspect, the techniques disclosed here feature an imageprocessing device including: an image input unit that inputs a facialimage from a predetermined device; an image analyzing unit thatcalculates one of facial shape, proportion lines that are lines drawn onthe face to analyze the balance of the face, and blocking lines thatdivide the face into multiple regions following the structure of theface according to lightness and darkness of shadows due to lightstriking the face, based on facial feature points extracted from thefacial image; and an image generating unit that decides a superimposingregion of a makeup parts image based on one of the facial shape, theproportion lines, and the blocking lines, and generates a simulationimage where the makeup parts image has been superimposed on thesuperimposing region.

According to the present disclosure, trouble and time for the instructorto explain makeup methods can be reduced, and students can obtain adeeper understanding of makeup methods.

It should be noted that general or specific embodiments may beimplemented as a system, a device, a method, an integrated circuit, acomputer program, a storage medium, or any selective combination ofsystem, device, method, integrated circuit, computer program, andstorage medium.

Additional benefits and advantages of the disclosed embodiments willbecome apparent from the specification and drawings. The benefits and/oradvantages may be individually obtained by the various embodiments andfeatures of the specification and drawings, which need not all beprovided in order to obtain one or more of such benefits and/oradvantages.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an example of a configuration ofan image processing device according to the present disclosure;

FIG. 2 is a diagram illustrating an example of facial feature points andfacial parts according to the present disclosure;

FIG. 3 is a flowchart illustrating a first operation example accordingto the present disclosure;

FIG. 4 is a diagram illustrating an example of proportion linesaccording to the present disclosure;

FIG. 5A is a diagram illustrating an example of superimposed regions ofcheek color images according to the present disclosure;

FIG. 5B is a diagram illustrating an example of superimposed regions oncheek color images according to the present disclosure;

FIG. 6 is a diagram illustrating another example of proportion linesaccording to the present disclosure;

FIG. 7 is a diagram illustrating another example of proportion linesaccording to the present disclosure;

FIG. 8A is a diagram illustrating another example of proportion linesaccording to the present disclosure;

FIG. 8B is a diagram illustrating another example of proportion linesaccording to the present disclosure;

FIG. 8C is a diagram illustrating another example of proportion linesaccording to the present disclosure;

FIG. 9 is a diagram illustrating a display example of simulation images,before averaging and after averaging, according to the presentdisclosure;

FIG. 10 is a flowchart illustrating a second operation example accordingto the present disclosure;

FIG. 11A is a diagram illustrating an example of matching facial shapes,according to the present disclosure;

FIG. 11B is a diagram illustrating an example of superimposed regions oflowlight images and highlight images for facial shapes, according to thepresent disclosure;

FIG. 12A is a diagram illustrating an example of matching facial shapes,according to the present disclosure;

FIG. 12B is a diagram illustrating an example of superimposed regions oflowlight images and highlight images for facial shapes, according to thepresent disclosure;

FIG. 13 is a flowchart illustrating a third operation example accordingto the present disclosure; and

FIG. 14 is a diagram illustrating an example of blocking lines accordingto the present disclosure.

DETAILED DESCRIPTION

An embodiment of the present disclosure will be described in detailbelow with reference to the drawings.

Configuration of Image Processing Device

First, the configuration of an image processing device 100 will bedescribed with reference to FIG. 1, FIG. 1 is a block diagramillustrating an example of the configuration of the image processingdevice 100.

The image processing device 100 may be a stationary device, or may be aportable device that can be easily carried about. The image processingdevice 100, a storage device 200, and a display device 300, may beprovided in, for example, a smartphone, a tablet terminal, a personalcomputer, or the like.

In the present embodiment, description will be made regarding an exampleof a case where the user of the image processing device 100 is aninstructor teaching makeup methods in a beautician school or the like,or a student learning makeup methods in a beautician school or the like,for example. The image processing device 100 has an image input unit101, an image analyzing unit 102, and an image generating unit 103, asillustrated in FIG. 1.

The image processing device 100 includes, for example, a centralprocessing unit (CPU), storage media such as read only memory (ROM) orthe like storing control programs, work memory such as random accessmemory (RAM) or the like, and a communication circuit, although omittedfrom illustration. The functions of the image input unit 101, imageanalyzing unit 102, and image generating unit 103 are realized by theCPU executing control programs, in this case.

The image input unit 101 inputs an image of a human face (hereinafterreferred to as facial image) from a predetermined device (omitted fromillustration), and outputs the facial image to the image analyzing unit102. The predetermined device may be a camera, or may be the storagedevice 200, for example. The facial image may be a moving image or maybe a still image. Note that the face in the facial image preferably is aface wearing no makeup.

The image analyzing unit 102 receives the facial image from the imageinput unit 101, and extracts facial feature points from the facialimage. The image analyzing unit 102 also extracts facial parts (e.g.,eyes, eyelids, cheeks, nose, lips, forehead, chin, and so forth), basedon the facial feature points.

Now, an example of facial feature points and facial parts will bedescribed with reference to FIG. 2. As illustrated in FIG. 2 forexample, multiple facial feature points (round dots in FIG. 2) areextracted from a facial image. For example, facial feature points 701through 704 make up the right eye. Accordingly, the image analyzing unit102 extracts a region 705 surrounded by the facial feature points 701through 704 as being the right eye (example of facial part). Also, forexample, facial feature points 701 through 703 and 706 through 708 makeup the right eyelid. Accordingly, the image analyzing unit 102 extractsa region 709 surrounded by the facial feature points 701 through 703 and706 through 708 as being the right eyelid (example of facial part).

Note that coordinates of the facial feature points are coordinates in afacial coordinate system set with multiple facial feature points as areference, for example. The coordinates may be either two-dimensionalcoordinates or three-dimensional coordinates. This so far has been adescription of an example of facial feature points and facial parts.

Returning to FIG. 1, the image analyzing unit 102 calculates, based onthe facial feature points, either proportion lines that are lines drawnon the face to analyze the facial shape (shape of the outline of thefrontal face), the balance of the face (balance of the overall face orbalance of facial parts), or blocking lines that divide the face intomultiple regions following the structure of the face (blocking),according to lightness and darkness of shadows due to light striking theface. Specific examples of facial shapes, proportion lines, and blockinglines will be described later in operation examples.

The image analyzing unit 102 then outputs the facial image, and analysisresults information indicating analysis results of the facial image, tothe image generating unit 103. Examples of analysis results informationinclude information indicating the type of each facial part that thathas been extracted, and information indicating coordinates of facialfeature points enabling identification of each facial part that has beenextracted. Examples of analysis results information also includeinformation indicating coordinates of facial feature points enablingidentification of facial shape, proportion lines, or blocking lines.

The image generating unit 103 receives the facial image and analysisresults information from the image analyzing unit 102, and generates asimulation image where a makeup parts image is superimposed on thefacial image, based on the analysis results information. The makeupparts image is an image indicating makeup regions and contrast, in orderto overlay the facial image with makeup items (e.g., eye shadow, cheekcolor, concealer, lipstick, highlight, lowlight, etc.) of predeterminedcolors for performing makeup simulation. The makeup parts image isstored in the storage device 200, and read out from the storage device200 by the image generating unit 103. Note that an arrangement may bemade where the storage device 200 stores multiple types of makeup partsimages with different colors and shapes for each makeup item, with theimage generating unit 103 reading out makeup parts images specified bythe user from these.

The image generating unit 103 then outputs the simulation image to thedisplay device 300. The display device 300 displays the simulationimage. Note that the facial image in the displayed simulation image maybe a facial image such as that in a normal mirror, or may be a facialimage in a flip mirror (horizontally inverting mirror). Examples ofgenerating processing of simulation images will be described later inoperation examples. The configuration of the image processing device 100has been described so far.

Operations of Image Processing Device

Next, operations of the image processing device 100 will be described.First through third operation examples will each be described below.

First Operation Example

First, a first operation example of the image processing device 100 willbe described with reference to FIG. 3. FIG. 3 is a flowchartillustrating the first operation example of the image processing device100.

First, the image input unit 101 inputs a facial image from apredetermined device (e.g., camera) (step S101). The image input unit101 then outputs the facial image to the image analyzing unit 102.

Next, the image analyzing unit 102 extracts facial feature points fromthe facial image received from the image input unit 101 (step S102). Theimage analyzing unit 102 also extracts facial parts based on the facialfeature points.

The image analyzing unit 102 calculates proportion lines next, based onthe facial feature points (step S103).

An example of proportion lines will be described here with reference toFIG. 4. For example, the image analyzing unit 102 calculates proportionlines L1 through L4, as illustrated in FIG. 4. The proportion lines L1through L4 are straight lines horizontal to the lateral width of theface, and are lines that divide the facial image in the verticaldirection. The proportion lines L1 through L4 are proportion lines usedto analyze the balance of the overall face.

The proportion line L1 is a straight line passing through facial featurepoints at the hairline on the forehead. The proportion line L2 is astraight line passing through facial feature points at the inner ends ofthe eyebrows. The proportion line L3 is a straight line passing throughfacial feature points at the nose tip. The proportion line L4 is astraight line passing through facial feature points at the tip of thechin.

The image analyzing unit 102 also calculates the width (distance)between proportion lines. For example, the image analyzing unit 102calculates a width w1 between the proportion line L1 and the proportionline L2, a width w2 between the proportion line L2 and the proportionline L3, and a width w3 between the proportion line L3 and theproportion line L4, as illustrated in FIG. 4. An example of proportionlines has been described so far.

Returning to FIG. 3, next, the image analyzing unit 102 outputs thefacial image and analysis results information to the image generatingunit 103. This analysis results information includes, for example,information such as the types of the facial parts, coordinates of facialfeature points enabling identification of the facial parts, coordinatesof facial feature points enabling identification of the proportion linesL1 through L4, the widths w1 through w3 between the proportion lines,and so forth.

Next, the image generating unit 103 generates an image indicating theproportion lines based on the analysis results information (hereinafterreferred to as proportion line image), and generates a simulation imagewhere the proportion line image has been superimposed on the facialimage (step S104). The image generating unit 103 then outputs thesimulation image to the display device 300 (step S104). For example, theproportion line image is an image of each of the proportion lines L1through L4 illustrated in FIG. 4.

The display device 300 displays the simulation image received from theimage generating unit 103. Accordingly, the user can view the proportionline image on the facial image.

Next, the image generating unit 103 decides superimposing regions ofmakeup parts images (cheek color images here as an example) on thefacial image, based on the analysis results information (Step 3105).

An example of superimposing region deciding processing (step S105) willbe described here. For example, the image generating unit 103 determineswhich of the width w1 and width w3 illustrated in FIG. 4 is longer.

In a case where width w1 is longer, cheek color image superimposingregions CH1 and CH2 on the facial image are decided so that the cheekcolor images are located upwards (with one end of the cheek color imagesnear the outer ends of the eyes and the other end far from the innerends of the eyes), as illustrated in FIG. 5A. On the other hand, in acase where width w3 is longer, cheek color image superimposing regionsCH3 and CH4 on the facial image are decided so that the cheek colorimages are located downwards (with one end of the cheek color imagesnear the inner ends of the eyes and the other end far from the outerends of the eyes), as illustrated in FIG. 5B. An example ofsuperimposing region deciding processing has been described so far.

Returning to FIG. 3, next, the image generating unit 103 reads out thecheek color images (e.g., images indicating pink ellipses of cheekcolor) from the storage device 200. Next, the image generating unit 103adjusts the sizes and shapes of the cheek color images in accordancewith the superimposing regions, and generates a simulation image wherethe cheek images have been superimposed on the superimposing regions(step S106). The image generating unit 103 then outputs the simulationimage to the display device 300 (step 3106).

The display device 300 displays the simulation image received from theimage generating unit 103. Accordingly, the user can visually recognizethe cheek images on the facial image. Note that the display device 300may display the simulation image generated in step S106, instead of thesimulation image generated in step S104. In this case, the proportionline image is erased from the facial image, and only the cheek colorimage is displayed superimposed on the facial image. Alternatively, thedisplay device 300 may display the cheek color image superimposed on thesimulation image generated in step S104. In this case, both theproportion line image and the cheek color image are displayedsuperimposed on the facial image.

The first operation example of the image processing device 100 has beendescribed so far. A modification of the first operation example will bedescribed below.

First Modification of First Operation Example

The processing of step S104 in FIG. 3 may be omitted. That is to say,superimposed display of the proportion line image does not have to beperformed.

Second Modification of First Operation Example

The processing of step S104 in FIG. 3 may be performed concurrently withthe processing of step S106. That is to say, a simulation image wherethe proportion line image and the makeup parts image (e.g., cheek colorimage) are both superimposed on the facial image may be generated andoutput to the display device 300.

Third Modification of First Operation Example

Although description has been made above by way of an example where themakeup parts image is a cheek color image, the makeup parts image may bea makeup parts image other than cheek color, such as the above-describedeye shadow, or the like, for example.

Fourth Modification of First Operation Example

The proportion lines that are calculated are not restricted to theproportion lines L1 through L4 illustrated in FIG. 4. Another example ofproportion lines will be described below.

First, proportion lines L5 through L8 will be described with referenceto FIG. 6. The proportion lines L5 through L8 illustrated in FIG. 6 areproportion lines used for analyzing the balance of the overall face. Theproportion lines L5 through L8 are straight lines that are even with thevertical width of the face, and are lines that divide the facial imagein the lateral direction.

The proportion line L5 is a straight line passing through facial featurepoints at the outer end of the right eye. The proportion line L6 is astraight line passing through facial feature points at the inner end ofthe right eye. The proportion line L7 is a straight line passing throughfacial feature points at the inner end of the left eye. The proportionline L8 is a straight line passing through facial feature points at theouter end of the left eye. The image analyzing unit 102 also calculatesa width w4 between the proportion line L5 and the proportion line L6, awidth w5 between the proportion line L6 and the proportion line L7, anda width w6 between the proportion line L7 and the proportion line L8.

Next, proportion lines L9 and L10 will be described with reference toFIG. 6. The proportion lines L9 and L10 illustrated in FIG. 6 areproportion lines used for analyzing the balance of the overall face. Theproportion lines L9 and L10 are straight lines that are horizontal withthe lateral width of the face, and are lines that divide the facialimage in the vertical direction. The proportion line L9 is a straightline passing through facial feature points at the upper lip. Theproportion line L10 is a straight line passing through facial featurepoints at the lower lip. The image analyzing unit 102 also calculates awidth w7 between the proportion line L3 and the proportion line L9, awidth w8 between the proportion line L9 and the proportion line L4, awidth w9 between the proportion line L3 and the proportion line L10, anda width w10 between the proportion line L10 and the proportion line L4,for example.

Next, proportion lines L11 through L14 will be described with referenceto FIG. 7. Note that the proportion lines L1 through L4 illustrated inFIG. 7 have already been described with reference to FIG. 4, andaccording description thereof will be omitted here. The proportion linesL11 through L14 illustrated in FIG. 7 are proportion lines used foranalyzing the balance of the overall face.

The proportion line L11 is a straight line passing through facialfeature points at the ridge of the nose (e.g., root of the nose, dorsumof the nose, tip of the nose). The proportion lines L12 are straightlines passing through facial feature points at the outer ends of theeyebrows, facial feature points at the outer ends of the eyes, andfacial feature points to the outer side of the wings of the nose. Theproportion line L13 is a straight line passing through facial featurepoints at the upper end of the right pupil and facial feature points atthe upper end of the left pupil. The proportion lines L14 are straightlines passing through an intersection point of the proportion line L11and proportion line L13, and facial feature points at the outer side ofthe wings of the nose.

Next, proportion lines L15 through L29 will be described with referenceto FIGS. 8A through 8C. The proportion lines L15 through L17 illustratedin FIG. 8A are proportion lines used for analyzing the balance of theeyebrow (example of a facial part). The proportion lines L15 through L17are straight lines that are even with the vertical width of the face.

In FIG. 8A, the proportion line L15 is a straight line passing throughfacial feature points at the outer end of the right eyebrow. Theproportion line L16 is a straight line passing through facial featurepoints at the peak of the arch of the right eyebrow. The proportion lineL17 is a straight line passing through facial feature points at theinner end of the right eyebrow. The image analyzing unit 102 alsocalculates a width w11 between the proportion line L15 and theproportion line L16, and a width w12 between the proportion line L16 andthe proportion line L17.

The proportion lines L18 through L23 illustrated in FIG. 8B areproportion lines used for analyzing the balance of the eye (example of afacial part). The proportion lines L18 through L21 are straight linesthat are even with the vertical width of the face, and the proportionlines L22 and L23 are straight lines that are horizontal with thelateral width of the face. In FIG. 8B, the proportion line L18 is astraight line passing through facial feature points at the outer end ofthe right eye. That is to say, the proportion line L18 is the same asthe proportion line L5 illustrated in FIG. 6. The proportion line L19 isa straight line passing through facial feature points at the right edgeof the pupil of the right eye. The proportion line L20 is a straightline passing through facial feature points at the left edge of the pupilof the right eye. The proportion line L21 is a straight line passingthrough facial feature points at the inner end of the right eye. That isto say, the proportion line L21 is the same as the proportion line L6illustrated in FIG. 6. The proportion line L22 is a straight linepassing through facial feature points at the lower edge of the rightpupil. The proportion line L23 is a straight line passing through facialfeature points at the upper edge of the right pupil.

The image analyzing unit 102 also calculates a width w13 between theproportion line L18 and the proportion line L19, a width w14 between theproportion line L19 and the proportion line L20, a width w15 between theproportion line L20 and the proportion line L21, and a width w16 betweenthe proportion line L22 and the proportion line L23.

The proportion lines L24 through L29 illustrated in FIG. 8C areproportion lines used for analyzing the balance of the mouth (example ofa facial part). The proportion lines L24 through L26 are straight linesthat are even with the vertical width of the face, and the proportionlines L27 through L29 are straight lines that are horizontal with thelateral width of the face. In FIG. 8C, the proportion line L24 is astraight line passing through facial feature points at the tip of thenose and facial feature points at the center of the lips. The proportionline L25 is a straight line passing through facial feature points at thecenter of the left nostril. The proportion line L26 is a straight linepassing through facial feature points at the left edge of the lips. Theproportion line L27 is a straight line passing through the lower edge ofthe lower lip. The proportion line L28 is a straight line passingthrough facial feature points at the boundary between the upper lip andlower lip. The proportion line L29 is a straight line passing throughthe upper edge of the upper lip.

The image analyzing unit 102 also calculates a width w17 between theproportion line L24 and the proportion line L25, a width w18 between theproportion line L25 and the proportion line L26, a width w19 between theproportion line L27 and the proportion line L28, and a width w20 betweenthe proportion line L28 and the proportion line L29.

Fifth Modification of First Operation Example

The positions of the above-described proportion lines L1 through L29 maybe changed based on user instructions, for example. Specifically, anarrangement may be made where, in a case of having received aninstruction to change the position of a predetermined proportion line,the image generating unit 103 generates a facial image where at leastone of the shape and position of a facial part in the facial image hasbeen changed based on the position of a proportion line after changing(hereinafter referred to as post-change facial image), and outputs thepost-change facial image to the display device 300. Alternatively, theimage generating unit 103 may generate a simulation image where aproportion line image is superimposed on the post-change facial image,and output to the display device 300.

Also, an arrangement may be made where, in a case of having received aninstruction to equalize the facial balance in the facial image(hereinafter, also referred to simply as “equalization”), the imagegenerating unit 103 changes the position of a predetermined proportionline so that the widths between the proportion lines have the sameratios.

For example, in a case of having received an instruction forequalization after having calculated the proportion lines L1 through L4illustrated in FIG. 4 and the proportion lines L5 through L8 illustratedin FIG. 6, the image generating unit 103 changes the position of atleast one of the proportion lines L1 through L4 so that the ratios ofthe widths w1 through w3 in FIG. 4 are 1:1:1, and changes the positionof at least one of the proportion lines L5 through L8 so that the ratiosof the widths w4 through w6 in FIG. 6 are 1:1:1.

Now, FIG. 9 illustrates a display example of a simulation image obtainedas a result of the above-described equalization. A simulation image Aillustrated in FIG. 9 is an example of a simulation image beforeequalization. On the other hand, a simulation image B illustrated inFIG. 9 is an example of a simulation image after equalization. Thesimulation images A and B are images where the proportion lines L1through L8 have been superimposed on a facial image, as illustrated inFIG. 9.

In comparison with the simulation image A, the simulation image B hashad the positions of the proportion lines L2 and L3 changed, and thepositions of the eyebrows and eyes have been changed. Thus, displayingtwo simulation images side by side allows easy comparison by the userbetween a face where the facial balance has not been equalized and aface where the facial balance has been equalized.

Although equalization processing where the positions of proportion linesare changed so that the widths have the same ratio (e.g., 1:1:1) hasbeen described in the above description, idealization processing may beperformed, for example, where the positions of proportion lines arehanged so that the ratio of the widths is an ideal ratio that has beenset beforehand. Examples of changing the position of proportion lines ina case of having received an instruction for idealization of facialbalance in a facial image will each be described below.

For example, the position of at least one of proportion lines L3, L9,and L4 in FIG. 6 may be changed so that the ratio of the width w7 andwidth w8 is 1:3. Also, the position of at least one of proportion linesL3, L10, and L4 in FIG. 6 may be changed so that the ratio of the widthw9 and width w10 is 1:1.

Also, for example, the position of at least one of proportion lines L15through L17 in FIG. 8A may be changed so that the ratio of the width WI1 and width w12 is 1:2. Also, for example, the position of at least oneof proportion lines L18 through L23 in FIG. 8B may be changed so thatthe ratio of the widths w13 through W16 1:1:1:1. Also, for example, theposition of at least one of proportion lines L24 through L26 in FIG. 8Cmay be changed so that the ratio of the width w17 and width w18 is 1:2.Also, for example, the position of at least one of proportion lines L27through L29 in FIG. 8C may be changed so that the ratio of the width w19and width w20 is 1.5:1.

Examples of changing the positions of proportion lines in a case ofidealization having been instructed has thus been described. In theseexamples as well, a post-change facial image where at least one of theposition and shape of a facial part has been change is generated basedon the changed proportion lines, as described above.

Sixth Modification of First Operation Example

An example has been described above regarding a case of decidingsuperimposing regions for cheek color images based on comparison resultsof width w1 and width w3, but this is not restrictive. Other exampleswill be described below.

For example, the image analyzing unit 102 calculates the vertical widthof the outline (e.g., may be proportion lines indicating the maximumvertical width) and lateral width of the outline (e.g., may beproportion lines indicating the maximum lateral width) based on facialfeature points, when performing the calculation processing of proportionlines (step S103 in FIG. 3). the calculation results thereof areincluded in the analysis results information and output to the imagegenerating unit 103.

The image generating unit 103 determines which of the calculatedvertical width and lateral width is longer, in the superimpositionregion deciding processing (step S105 in FIG. 3). In a case where thevertical width is longer (in a case where the facial shape is an oblongface), the superimposing region of the cheek color images on the facialimage is decided so that elliptical cheek color images are laid outsideways (so that the long axis of the ellipse is parallel to thelateral width of the face), for example. On the other hand, in a casewhere the lateral width is longer (in a case where the facial shape is around face), the superimposing region of the cheek color images on thefacial image is decided so that elliptical cheek color images are laidout obliquely from the cheeks toward the corners of the mouth, forexample. Modification of the first operation example have thus beendescribed.

Second Operation Example

Next, a second operation example of the image processing device 100 willbe described with reference to FIG. 10. FIG. 10 is a flowchartillustrating the second operation example of the image processing device100. Steps S101 and S102 have already been described in the firstoperation example, so description here will be omitted.

Next, the image analyzing unit 102 calculates the facial shape based onthe facial feature points (step S203). The facial shapes calculated here(hereinafter referred to as calculated facial shape) are, for example,triangular, diamond, round, oblong, rectangular, octagonal, and soforth.

Next, image analyzing unit 102 outputs the facial image and the analysisresults information to the image generating unit 103. The analysisresults information includes, for example, types of the facial parts,coordinates of facial feature points enabling identification of thefacial parts, coordinates of facial feature points enablingidentification of calculated facial shape, and so forth.

Next, the image generating unit 103 reads information of a standardfacial shape out from the storage device 200. The standard facial shapeis a facial shape decided beforehand, and is, for example, an oval shapethat is an ideal face form.

Next, the image generating unit 103 matches the calculated facial shapeand the standard facial shape by matching the size of the standardfacial shape to the size of the calculated facial shape, and calculatesoverspread regions and insufficient regions (step S204).

Next, the image generating unit 103 decides the superimposing region ofa makeup parts image (lowlight image and highlight image here, as anexample) to be superimposed on the facial image, based on the overspreadregions and insufficient regions (step S205).

Examples of region calculation processing (step S204) and superimposingregion deciding processing (step S205) will be described here, withreference to FIGS. 11A through 12B. Region calculation processing andsuperimposing region deciding processing in a case where the calculatedfacial shape is triangular will be described with reference to FIGS. 11Aand 11B.

First, the image generating unit 103 matches the triangular calculatedfacial shape t1 with the oval reference facial shape T, as illustratedin FIG. 11A. The image generating unit 103 then calculates overspreadregions X1 and X2 where the calculated facial shape t1 spreads out fromthe reference facial shape T. The image generating unit 103 alsocalculates insufficient regions Y1 and Y2 where the calculated facialshape t1 does not fill out the reference facial shape T.

Next, the image generating unit 103 decides regions LL1 and LL2corresponding to the overspread regions X1 and X2 of the facial image tobe lowlight image superimposing regions, as illustrated in FIG. 11B. Theimage generating unit 103 also decides regions HL1 and HL2 correspondingto the insufficient regions Y1 and Y2 of the facial image to behighlight image superimposing regions, as illustrated in FIG. 11B.

Also, as illustrated in FIG. 11B, the image generating unit 103 decidesa region HL3 (region of forehead and nose ridge), region HL4 (regionbelow the right eye), region HL5 (region below the left eye), and regionHL6 (region of the chin), as highlight image superimposing regions.These regions HL3 through HL6 are decided beforehand as highlight imagesuperimposing regions common to all face forms, but the sizes and shapesare changed as appropriate in accordance with features of the facialimage (calculated facial shape).

Region calculation processing and superimposing region decidingprocessing in a case where the calculated facial shape is a diamondshape will be described with reference to FIGS. 12A and 12B. First, theimage generating unit 103 matches the diamond-shaped calculated facialshape t2 with the oval reference facial shape T, as illustrated in FIG.12A. The image generating unit 103 then calculates overspread regions X3and X4 where the calculated facial shape t2 spreads out from thereference facial shape T. The image generating unit 103 also calculatesinsufficient regions Y3 and Y4 where the calculated facial shape t2 doesnot fill out the reference facial shape T.

Next, the image generating unit 103 decides regions LL3 and LL4corresponding to the overspread regions X3 and X4 of the facial image tobe lowlight image superimposing regions, as illustrated in FIG. 12B. Theimage generating unit 103 also decides regions HL7 and HL8 correspondingto the insufficient regions Y3 and Y4 of the facial image to behighlight image superimposing regions, as illustrated in FIG. 12B. Theimage generating unit 103 also decides regions HL3 through HL6 to behighlight image superimposing regions, as illustrated in FIG. 12B.Examples of region calculation processing and superimposing regiondeciding processing have thus been described.

Returning to FIG. 10, next, the image generating unit 103 reads outlowlight images and highlight images from the storage device 200. Theimage generating unit 103 then adjusts the sizes and shapes of thelowlight images and highlight images in accordance with thesuperimposing regions, and generates a simulation image where thelowlight images and highlight images have been superimposed on thesuperimposing regions (step S206). The image generating unit 103 outputsthe simulation image to the display device 300, next (Step S206).

The display device 300 displays the simulation image received from theimage generating unit 103. Accordingly, the user can visually recognizethe lowlight images and highlight images on the facial image. The secondoperation example of the image processing device 100 has thus beendescribed. Modifications of the second operation example will bedescribed next.

First Modification of Second Operation Example

Although description has been made above giving an example of a casewhere the makeup parts images are lowlight images and highlight images,The makeup parts images may be images of makeup items other thanlowlights and highlights.

Second Modification of Second Operation Example

Although description has been made above giving an example of a casewhere the calculated facial shape is triangular or diamond-shaped, thecalculated facial shape may be round, oblong, rectangular, or octagonal.

Third Modification of Second Operation Example

Although description has been made above giving an example of a casewhere both overspread regions and insufficient regions are calculated,there may be cases where only one of overspread regions and insufficientregions is calculated, depending on the matching results.

Also, even in a case where both overspread regions and insufficientregions are calculated, there is no need to decide all calculatedregions to be superimposing regions. For example, in cases wheresuperimposing highlight images in regions corresponding to insufficientregions in the facial image results in an unnatural appearance, suchinsufficient regions are not decided to be superimposing regions.Modifications of the second operation example have thus been described.

Third Operation Example

Next, a third operation example of the image processing device 100 willbe described with reference to FIG. 13. FIG. 13 is a flowchartillustrating the third operation example of the image processing device100. Steps S101 and S102 have already been described in the firstoperation example, so description here will be omitted.

Next, the image analyzing unit 102 calculates secondary feature pointsbased on the facial feature points (step S303). Secondary feature pointsare, for example, points dividing line segments between facial featurepoints (hereinafter referred to simply as “line segment”), points onextensions of line segments, intersection points of two line segments,and so forth.

Next, the image analyzing unit 102 calculates blocking lines based onthe facial feature points and secondary feature points (step S304).

Next, the image analyzing unit 102 calculates the luminances of dividedregions sectioned by the blocking lines (step S305).

An example of blocking lines and divided regions will be described herewith reference to FIG. 14. For example, the image analyzing unit 102calculates blocking lines BL, as illustrated in FIG. 14. The round dotsin FIG. 14 represent facial feature points, and the squares representsecondary feature points. Also, a divided region DA indicates one ofmultiple divided regions sectioned by the blocking lines BL in FIG. 14.An example of blocking lines and divided regions has thus beendescribed.

Returning to FIG. 13, next, the image analyzing unit 102 outputs thefacial image and analysis results information to the image generatingunit 103. The analysis results information includes, for example, typesof the facial parts, coordinates of facial feature points enablingidentification of the facial parts, coordinates of facial feature pointsand secondary feature points enabling identification of blocking linesBL, coordinates of facial feature points and secondary feature pointsenabling identification of divided regions, luminance of each dividedregion, and so forth.

Next, the image generating unit 103 generates an image indicatingblocking lines (hereinafter referred to as blocking line image) based onthe analysis results information, and generates a simulation image wherethe blocking line image has been superimposed on the facial image (stepS306). The image generating unit 103 outputs the simulation image to thedisplay device 300 (step S306). For example, the blocking line image isan image showing the blocking lines BL illustrated in FIG. 14. Note thatfacial feature points and secondary feature points may be included inthis blocking line image.

The display device 300 displays the simulation image received from theimage generating unit 103. Accordingly, the user can visually recognizethe blocking line image on the facial image.

Next, the image generating unit 103 decides superimposing regions of themakeup parts image on the facial image (lowlight images and highlightimages here as an example) based on the analysis results information(step S307).

An example of superimposing region deciding processing (step S307) willbe described here. First, the image generating unit 103 determineswhether or not the luminance of each divided region is equal to orgreater than a first threshold value that has been set beforehand. Theimage generating unit 103 then decides divided regions where theluminance is greater than the first threshold value to be highlightimage superimposing regions.

On the other hand, if the luminance is smaller than the first thresholdvalue, the image generating unit 103 determines whether or not theluminance smaller than a second threshold value. The second thresholdvalue is a smaller value than the first threshold value. The imagegenerating unit 103 then decides divided regions where the luminance issmaller than the second threshold value to be lowlight imagesuperimposing regions.

On the other hand, the image generating unit 103 sets divided regionsthat are smaller than the first threshold value but equal to or largerthan the second threshold value to be non-superimposing regions.Non-superimposing regions are regions where neither highlight images norlowlight images are superimposed. An example of superimposing regiondeciding processing has thus been described.

Returning to FIG. 13, next, the image generating unit 103 reads outlowlight images and highlight images from the storage device 200. Theimage generating unit 103 then adjusts the sizes and shapes of thelowlight images and highlight images to match the superimposing regions,and generates a simulation image where the lowlight images and highlightimages have been superimposed on the superimposing regions (step S308).The image generating unit 103 then outputs the simulation image to thedisplay device 300 (step S308).

The display device 300 displays the simulation image received from theimage generating unit 103. Accordingly, the user can visually recognizelowlight images and highlight images on the facial image. The thirdoperation example of the image processing device 100 has thus beendescribed. Modifications of the third operation example will bedescribed below.

First Modification of Third Operation Example

Although description has been made above regarding an example of a casewhere the makeup parts image is lowlight images and highlight images,the makeup parts image may be makeup items other than lowlights andhighlights.

Second Modification of Third Operation Example

Although description has been made above regarding an example of a casewhere blocking lines are calculated based on facial feature points andsecondary feature points, blocking lines may be calculated based onfacial feature points alone. Modifications of the third operationexample, and operations of the image processing device 100, have thusbeen described.

Advantages of Present Embodiment

As described above, the image processing device 100 according to thepresent embodiment is a device that generates a simulation image formakeup. The image processing device 100 includes the image input unit101 that inputs a facial image from a predetermined device, the imageanalyzing unit 102 that calculates one of facial shape, proportion linesthat are lines drawn on the face to analyze the balance of the face, andblocking lines that divide the face into multiple regions following thestructure of the face according to lightness and darkness of shadows dueto light striking the face, based on facial feature points extractedfrom the facial image, and the image generating unit 103 that generatesa simulation image where a makeup parts image has been superimposed inthe facial image based on one of the facial shape, proportion lines andblocking lines, and outputs the simulation image to the display device300. Thus, according to the present embodiment, trouble and time for theinstructor to explain makeup methods can be reduced, and students canobtain a deeper understanding of makeup methods.

MODIFICATIONS OF PRESENT DISCLOSURE

An embodiment of the present disclosure has been described above, butthe present disclosure is not restricted to the above description, andvarious modification can be made. Modifications will be described below.

First Modification

In a case where a user selects a makeup parts image included in asimulation image during display of the simulation image, the imageprocessing device 100 may output a method and so forth of applying thatmakeup item, to the display device 300 as a makeup guide imagesuperimposed on the facial image, or guide information in the form oftext, photographs, illustrations, audio, or the like. Guide informationis stored in the storage device 200 in correlation with each makeupparts image. Note that the guide information may be still images ormoving images. The display device 300 may have an audio output (speaker)function (which is true for the following modifications as well).

For example, when generating a simulation image, the image generatingunit 103 reads out from the storage device 200 makeup parts images, andalong therewith reads out guide information correlated with the makeupparts images. Thereafter, in a case where an operation for selecting amakeup parts image is performed by the user while the simulation imageis displayed, the image generating unit 103 outputs guide informationcorresponding to the selected makeup parts image to the display device300. The display device 300 then displays the guide information.Accordingly, the user can readily comprehend the method and so forth ofapplying makeup items.

Second Modification

The image processing device 100 may input a facial image indicating aface where makeup has actually been applied following a simulation imageon which a makeup parts image has been superimposed, and evaluate theactually-applied makeup by comparing this facial image with thesimulation image.

For example, the user actually applies makeup to his/her own face whileviewing the simulation image on which the makeup parts image has beensuperimposed (hereinafter referred to as model image), and photographsthe face with a camera. The image input unit 101 then inputs the facialimage from the camera (hereinafter referred to as makeup-completedimage), and outputs the makeup-completed image to the image analyzingunit 102.

Next, the image analyzing unit 102 extracts regions where makeup hasactually been applied (hereinafter referred to as makeup regions) fromthe makeup-completed image. The image analyzing unit 102 then outputsthe makeup-completed image and analysis results information includingcoordinates and so forth enabling identification of the makeup regions,to the image generating unit 103.

Next, the image generating unit 103 matches the makeup regions indicatedin the analysis results information (e.g., regions where cheek color hasactually been applied), and makeup parts image superimposing regionsincluded in the model image (e.g., cheek color image superimposingregions). The image generating unit 103 then calculates the proportionof matching between the makeup regions and superimposing regions(hereinafter referred to as evaluation value). The higher the value ofthe evaluation value is, this indicates that the greater the accuracy ofmakeup is.

The image generating unit 103 generates information indicating theevaluation value (hereinafter referred to as makeup evaluationinformation), and outputs to the display device 300. The display device300 displays the makeup evaluation information. Accordingly, the usercan comprehend to what degree the makeup has been accurately performed.

Although an example of a case where the makeup evaluation information isan evaluation value has been described above, this is not restrictive.For example, the makeup evaluation information may be a message,illustration, or the like indicating the accuracy of the makeup, decidedbase on the evaluation value.

Although an example of a case where the makeup is evaluated by matchingthe makeup regions and superimposing regions has been described above,this is not restrictive. For example, evaluation of makeup may beperformed by matching the color of makeup regions and the color of themakeup parts image. In a case where the results of matching show thatthere are portions where color difference is a predetermined value ofhigher (e.g., portions with uneven coloring), the evaluation value maybe calculated to be lower. Also, the makeup evaluation information mayinclude information indicating the makeup region with uneven coloring inthe form of text, arrows, audio, or the like.

Third Modification

The image processing device 100 may extract skin color from the facialimage and select a makeup parts image to be superimposed on the facialimage in accordance with that skin color. For example, the imageanalyzing unit 102 extracts the color of skin regions (e.g., cheek,forehead, chin, etc.) from the facial image. The image analyzing unit102 then outputs analysis results information including information ofthe color of skin regions to the image generating unit 103.

The image generating unit 103 reads out makeup parts images (e.g.,concealer images, foundation images, etc.) having the same color as thecolor of the skin regions indicated in the analysis results information,from the storage device 200. The image generating unit 103 then adjuststhe shapes and sizes of the makeup parts images to match thesuperimposing regions, generates a simulation image where the makeupparts images are superimposed on the facial image, and outputs thesimulation image to the display device 300. The display device 300displays this simulation image. Accordingly, the image processing device100 can propose to the user a makeup parts image suitable for actualskin color.

Although description has been made above regarding an example of a casewhere the image generating unit 103 selects makeup parts images havingthe same color as the color of the skin regions, this is notrestrictive. For example, the image generating unit 103 may selectmakeup parts images of approximate colors to the color of the skinregions. For example, an approximate color may be a color that is one ortwo steps brighter or a color that is one or two steps darker than thecolor of the skin regions.

Fourth Modification

In a case of having selected one makeup parts image (this may be amakeup parts image specified by the user) to be superimposed on a facialimage, the image processing device 100 may select another makeup partsimage based on the color of that makeup parts image.

For example, the storage device 200 stores at least one or more makeupparts images correlated with one makeup parts image. This correlation isset beforehand so that the color harmony is achieved among the makeupparts images, based on the Munsell color system, for example. In a casewhere a foundation image of a predetermined color has been selected as amakeup parts image to be superimposed on a facial image and has beenread out from the storage device 200, for example, the image generatingunit 103 reads out, from the storage device 200, cheek color images, eyeshadow images, and so forth, of predetermined colors correlated with thefoundation image. The image generating unit 103 then adjusts the shapesand sizes of these makeup parts images to match the superimposingregions, generates a simulation image where the makeup parts images aresuperimposed on the facial image, and outputs the simulation image tothe display device 300. The display device 300 displays this simulationimage. Accordingly, the image processing device 100 can propose multiplemakeup parts images, with color harmony achieved, to the user.

Fifth Modification

In a case where a discoloration region is present in a skin region inthe facial image, the image processing device 100 may generate asimulation image where a makeup parts image to cover the discolorationregion has been superimposed on the facial image. Discoloration regionsinclude pigmented spots, chloasma, nevus spilus, melanocytic nevus,nevus of Ota, acquired dermal melanocytosis, erythema, purpura,vitiligo, bruises, moles, darkening of pores, sunburn areas, acne, acnescars, pigmentation due to friction or inflammation, wrinkles, freckles,tattoos, warts, scarring, and so forth.

For example, the image analyzing unit 102 extracts a region where colordifference is a predetermined value or higher as compared withsurrounding regions (regions in the skin region around the discolorationregion), as a discoloration region. The image analyzing unit 102 thenoutputs analysis results information including information such as thecolor of the surrounding regions, coordinates of facial feature pointsenabling identification of the discoloration region, and so forth, tothe image generating unit 103.

The image generating unit 103 reads out a makeup parts image having acolor of the surrounding regions indicated in the analysis resultsinformation (e.g., for blemish portions such as pigmented spots or thelike, a concealer image, foundation image, etc., approximating the skincolor or foundation actually applied) from the storage device 200. Theimage generating unit 103 then adjusts the shapes and sizes of themakeup parts images to match the discoloration region, generates asimulation image where the makeup parts image is superimposed on thefacial image, and outputs the simulation image to the display device300. The display device 300 displays this simulation image. Accordingly,the user can comprehend makeup items of a suitable color to coverdiscoloration regions.

Sixth Modification

In a case of having received a switching instruction from the user whiledisplaying a simulation image where the facial image is a still image,the image processing device 100 may switch the facial image from thestill image to a moving image. Accordingly, the user can visuallyrecognize simulation images of the face tilted to various directions,which makes it further easier to comprehend the colors, shapes,positions, and so forth, of makeup parts images.

Also, in a case of having received a switching instruction from the userwhile displaying a simulation image where the facial image is a movingimage, the image processing device 100 may switch the facial image fromthe moving image to a still image. Accordingly, the user can take timeto visually recognize a simulation image of the face tilted at apredetermined angle, which makes it further easier to comprehend thecolors, shapes, positions, and so forth, of makeup parts images.

Seventh Modification

The image processing device 100 may output multiple simulation images tothe display device 300, and the display device 300 may display themultiple simulation images side by side. The multiple simulation imagesbeing displayed side by side may be of the same type, or may be ofdifferent types. For example, a simulation image where a proportion lineimage has been superimposed on a predetermined facial image (see firstoperation example), and a simulation image where a proportion line imagehas been superimposed on a facial image other than the predeterminedfacial image (see first operation example), may be displayed side byside.

Also, a simulation image where a proportion line image has beensuperimposed on a predetermined facial image (see first operationexample), and a simulation image where a blocking lines image has beensuperimposed on the same facial image as the predetermined facial image(see third operation example), may be displayed side by side.Accordingly, the user can compare multiple simulation images morereadily.

Eighth Modification

The image processing device 100 (e.g., the image generating unit 103,same as above in the present modification) may output only the facialimage to the display device 300, and the display device 300 may displayjust the facial image. Further, in a case where the user has selected amakeup parts image during display of the facial image alone and asuperimposing instruction of that makeup parts image has been received,the image processing device 100 may generate a simulation image wherethe makeup parts image has been superimposed on the facial image beingdisplayed, and output to the display device 300.

Also, in a case of outputting just the facial image, the imageprocessing device 100 may also output a facial image indicating a rangeover which makeup parts images can be superimposed (hereinafter referredto as superimposable range) to the display device 300. Thesuperimposable range is, for example, out of ranges where a facial imagefacing forward has been vertically bisected, one of an image of theright half of the face (hereinafter referred to as right face image) andan image of the left half of the face (hereinafter referred to as leftface image). For example, in a case where the right face image is thesuperimposable range, the image processing device 100 generates a facialimage where the luminance of the left face image is lower than theluminance of the right face image, and outputs to the display device300. Thus, the user can recognize that the right face image is thesuperimposable range.

The student then, for example, selects a desired makeup parts image at aterminal that the image processing device 100 has, and performsoperations to instruct the makeup parts image to be superimposed at adesired position in the right face image. On the other hand, theinstructor selects a desired makeup parts image at a terminal that theimage processing device 100 has (which may be the same terminal that thestudent is using, or may be a separate terminal), and performsoperations to instruct the makeup parts image to be superimposed at adesired position in the left face image. Note that besides the method ofselecting a makeup parts image and instructing the superimposingposition thereof, the student and instructor may use an electronic penor the like, for example, to directly draw a makeup parts image on theleft face image displayed on the terminal.

Upon having accepted the above-described operations by the student andthe instructor, the image processing device 100 generates a simulationimage where the makeup parts image selected by the student issuperimposed on the right face image (hereinafter referred to as rightface simulation image). The image processing device 100 also generates asimulation image where the makeup parts image selected by the instructoris superimposed on the left face image (hereinafter referred to as leftface simulation image). The image processing device 100 then outputs theright face simulation image and left face simulation image to thedisplay device 300.

The display device 300 combines the right face simulation image and theleft face simulation image to display as a single facial simulationimage. At this time, the right face simulation image and the left facesimulation image are displayed at the same luminance. Note that theright face simulation image may be displayed with lower luminance thanthe left face simulation image, or the left face simulation image may bedisplayed with lower luminance than the right face simulation image.Accordingly, the student can visually recognize the right facesimulation image generated by his/her own operations and the left facesimulation image generated by operations of the instructor, and caneasily compare.

Note that the image processing device 100 may store the right facesimulation image and the left face simulation image in the storagedevice 200 as learning history. Simulation images to be stored in thestorage device 200 as learning history are not restricted to right facesimulation images and left face simulation image, and may be simulationimages of the entire face, for example.

The image processing device 100 may also perform image analysis ofmultiple right face simulation images stored as learning history, andevaluate tendencies of makeup by the student (e.g., the way that makeupparts images are laid out, color selection of makeup parts images,etc.). The image processing device 100 may output information indicatingresults of evaluation to the display device 300, and the display device300 may display this information. Accordingly, the student cancomprehend tendencies of his/her own makeup.

The image processing device 100 may also compare the right facesimulation image and the left face simulation image, and determinewhether the positions, colors, and so forth of makeup parts images thatthe student has selected agree with the positions, colors, and so forthof makeup parts image that the instructor has selected. The imageprocessing device 100 may also output the results of determination tothe display device 300 as determination results information indicated bytext, illustrations, audio, or the like. The display device 300 thendisplays the determination results information. Accordingly, the studentcan comprehend mistakes and so forth regarding positions, colors, and soforth, of the makeup parts images that he/she has selected.

Ninth Modification

Terminals having the image processing device 100 and display device 300may communicate with each other. For example, simulation images (e.g.,the right face simulation image and left face simulation image describedin the eighth modification) may be exchanged between a terminal that thestudent uses and a terminal that the instructor uses. Accordingly,students residing or staying at remote locations can receive instructionregarding makeup by an instructor.

Also, for example, the instructor creates course data beforehand,indicating the way to proceed with a textbook (e.g., speed of page feed,highlighting of important points, etc.) for remote learning(correspondence learning), and stores the course data in the terminal.The terminal that the student at a remote location uses receives thecourse data, and displays the textbook based on the course data.Accordingly, remote learning can be performed with a sense of presence,as if the student were actually in class. Note that the instructor mayin real time perform operations instructing how to proceed with thetextbook, or perform coaching operations regarding a simulation imagethat the student has created. The terminal that the student usesreceives information indicating these operations, and performs displayand so forth based on the information.

Tenth Modification

The method of extracting facial feature points and the method ofextracting facial parts are not restricted to the description in theabove-described embodiment. Known classification methods, patternrecognition methods, clustering methods, and optimization methods may beemployed.

Examples of known classification methods include decision tree analysis,neural networks (including deep learning), and naive Bayes. Examples ofknown pattern recognition methods include neural networks (includingdeep learning), and support vector machines (SVM). Examples of knownclustering methods include k-Nearest Neighbors (k-NN), k-means, andhierarchical clustering. Examples of known optimization methods includegenetic algorithms.

Eleventh Modification

Part of the configuration of the image processing device 100 may bephysically distanced from other parts of the configuration of thedevice. In this case, the multiple distanced parts each need to have acommunication unit to communicate with each other. For example, part ofthe functions of the image processing device 100 may be in the Cloud.The image processing device 100 may also include at least one of thestorage device 200 and display device 300. The image processing device100 may also include a device that outputs facial images to the imageinput unit 101 (e.g., a camera).

For example, in a case where a terminal having the image processingdevice 100 and display device 300 is connected to a network, createddata and operation history may be saved in the Cloud on the network,besides in the terminal. Information collected in the Cloud includesbasic facial shapes, facial feature point information, shapes of makeupparts that have been created, history of brush touches and ways ofapplication, information of colors used, information of makeup itemproducts, and so forth. Such information is learned in the Cloud, andtrends are analyzed over periods of months or periods of years. Facialshapes and makeup techniques that are trending in that age, such as lineshapes, cheek color techniques, and so forth, are analyzed from facialshapes, facial feature point information, shapes of makeup parts, andbrush operation history. Popular color usages and product informationare extracted from color information and makeup item productinformation.

Such learning data is accumulated as trending makeup methods and colorinformation, and is automatically downloaded to terminals. The newestmakeup information can constantly be referenced and used at theterminals, and guidelines for ways to perform makeup such as illustratedin FIGS. 5A, and 5B, and FIGS. 11A through 12B, for example, areautomatically updated to the newest makeup.

An image processing device according to the present disclosure includes:an image input unit that inputs a facial image from a predetermineddevice; an image analyzing unit that calculates one of facial shape,proportion lines that are lines drawn on the face to analyze the balanceof the face, and blocking lines that divide the face into multipleregions following the structure of the face according to lightness anddarkness of shadows due to light striking the face, based on facialfeature points extracted from the facial image; and an image generatingunit that decides a superimposing region of a makeup parts image basedon one of the facial shape, the proportion lines, and the blockinglines, and generates a simulation image where the makeup parts image hasbeen superimposed on the superimposing region.

Note that in the image processing device, the image generating unit maycalculate an overspread region where the facial shape spreads out from astandard facial shape that has been set beforehand, and an insufficientregion where the facial shape does not fill out the standard facialshape, decide a region of the facial image corresponding to theoverspreading region to be a superimposing region for a first makeupparts image, and decide a region of the facial image corresponding tothe insufficient region to be a superimposing region for a second makeupparts image that is different from the first makeup parts image.

Also, in the image processing device, the image generating unit maycalculate luminance of divided regions which the facial image has beendivided into by the blocking lines, decide a divided region, where theluminance in the facial image is greater than a first threshold value,to be a superimposing region for a second makeup parts image, and decidea divided region, where the luminance in the facial image is smallerthan a second threshold value that is smaller than the first thresholdvalue, to be a superimposing region for a first makeup parts image thatis different from the second makeup parts image.

Also, in the image processing device, the first makeup parts image maybe a lowlight image, and the second makeup parts image may be ahighlight image.

Also, in the image processing device, the image analyzing unit maycalculate a first proportion line, a second proportion line below thefirst proportion line, and a third proportion line below the secondproportion line, that divide the facial image in the vertical direction.The image generating unit may decide, in a case where a first distancebetween the first proportion line and the second proportion line islonger than a second distance between the second proportion line and thethird proportion line, a superimposing position of the makeup partsimage where the makeup parts image is laid out upwards, and decide, in acase where the second distance is longer than the first distance, asuperimposing position of the makeup parts image where the makeup partsimage is laid out downwards.

Also, in the image processing device, the image analyzing unit maycalculate a fourth proportion line indicating a maximum vertical widthof the facial image, and a fifth proportion line indicating a maximumlateral width of the facial image. The image generating unit may decide,in a case where the fourth proportion line is longer than the fifthproportion line, a superimposing position of the makeup parts imagewhere a longitudinal direction of the makeup parts image is laid outfollowing the lateral width of the facial image, and decide, in a casewhere the fifth proportion line is longer than the fourth proportionline, a superimposing position of the makeup parts image where themakeup parts image is laid out obliquely from a cheek toward a mouthcorner.

Also, in the image processing device, the makeup parts image may be acheek color image.

Also, in the image processing device, the image generating unit maygenerate the simulation image where an image of the proportion lines issuperimposed on the facial image.

Also, in the image processing device, the image generating unit maygenerate the simulation image where an image of the blocking lines issuperimposed on the facial image.

Also, in the image processing device, in a case of having received aninstruction to change positions of the proportion lines, the imagegenerating unit may generate a facial image where at least one of theform and position of a facial part in the facial image has been changedbased on the positions of the proportion lines after changing.

Also, in the image processing device, the image analyzing unit maycalculate at least three or more of the proportion lines. The imagegenerating unit may change, in a case of having received an instructionfor idealization of facial balance of the facial image, the positions ofthe proportion lines to where the distances between the proportion linesare at a ratio decided beforehand, and generate a facial image where atleast one of the form and position of a facial part in the facial imagehas been changed based on the positions of the proportion lines afterchanging.

Also, in the image processing device, the image generating unit mayoutput the simulation image to a predetermined display device.

An image processing method according to the present disclosure includes:inputting a facial image from a predetermined device; calculating one offacial shape, proportion lines that are lines drawn on the face toanalyze the balance of the face, and blocking lines that divide the faceinto multiple regions following the structure of the face according tolightness and darkness of shadows due to light striking the face, basedon facial feature points extracted from the facial image; deciding asuperimposing region of a makeup parts image in the facial image, basedon one of the facial shape, the proportion lines, and the blockinglines; and generating a simulation image where the makeup parts imagehas been superimposed on the superimposing region.

An image processing program according to the present disclosure causes acomputer to execute processing of inputting a facial image from apredetermined device, processing of calculating one of facial shape,proportion lines that are lines drawn on the face to analyze the balanceof the face, and blocking lines that divide the face into multipleregions following the structure of the face according to lightness anddarkness of shadows due to light striking the face, based on facialfeature points extracted from the facial image, processing of deciding asuperimposing region of a makeup parts image in the facial image, basedon one of the facial shape, the proportion lines, and the blockinglines, and processing of generating a simulation image where the makeupparts image has been superimposed on the superimposing region.

The image processing device, image processing method, and imageprocessing program according to the present disclosure are useful as animage processing device, image processing method, and image processingprogram that generate simulation images for makeup.

What is claimed is:
 1. An image processing device comprising: an imageinput unit that inputs a facial image from a predetermined device; animage analyzer that calculates one of a facial shape, proportion linesthat are lines drawn on the face to analyze the balance of the face, andblocking lines that divide the face into multiple regions following thestructure of the face according to lightness and darkness of shadowsarising due to light striking the face, based on facial feature pointsextracted from the facial image; and an image generator that determinesa superimposing region of a makeup parts image based on one of thefacial shape, the proportion lines, and the blocking lines, andgenerates a simulation image where the makeup parts image has beensuperimposed on the superimposing region wherein the image generatorcalculates an overspread region where the facial shape spreads out froma standard facial shape that has been set beforehand, and aninsufficient region where the facial shape does not fill out thestandard facial shape, determines a region of the facial imagecorresponding to the overspreading region to be a superimposing regionfor a first makeup parts image, and determines a region of the facialimage corresponding to the insufficient region to be a superimposingregion for a second makeup parts image that is different from the firstmakeup parts image.
 2. The image processing device according to claim 1,wherein the first makeup parts image is a lowlight image, and whereinthe second makeup parts image is a highlight image.
 3. The imageprocessing device according to claim 1, wherein the image analyzercalculates a first proportion line, a second proportion line below thefirst proportion line, and a third proportion line below the secondproportion line, that divide the facial image in the vertical direction,and wherein the image generator determines, in a case where a firstdistance between the first proportion line and the second proportionline is longer than a second distance between the second proportion lineand the third proportion line, a superimposing position of the makeupparts image where the makeup parts image is laid out upwards, anddetermines, in a case where the second distance is longer than the firstdistance, a superimposing position of the makeup parts image where themakeup parts image is laid out downwards.
 4. The image processing deviceaccording to claim 3, wherein the makeup parts image is a cheek colorimage.
 5. The image processing device according to claim 1, wherein theimage analyzer calculates a fourth proportion line indicating a maximumvertical width of the facial image, and a fifth proportion lineindicating a maximum lateral width of the facial image, and wherein theimage generator determines, in a case where the fourth proportion lineis longer than the fifth proportion line, a superimposing position ofthe makeup parts image where a longitudinal direction of the makeupparts image is laid out following the lateral width of the facial image,and determines, in a case where the fifth proportion line is longer thanthe fourth proportion line, a superimposing position of the makeup partsimage where the makeup parts image is laid out obliquely from a cheektoward a mouth corner.
 6. The image processing device according to claim1, wherein the image generator generates the simulation image where animage of the proportion lines is superimposed on the facial image andcalculates the proportion lines.
 7. The image processing deviceaccording to claim 1, wherein the image generator generates thesimulation image where an image of the blocking lines is superimposed onthe facial image and calculates the blocking lines.
 8. The imageprocessing device according to claim 1, wherein, in a case of havingreceived an instruction to change positions of the proportion lines, theimage generator generates a facial image where at least one of the formand position of a facial part in the facial image has been changed basedon the positions of the proportion lines after changing and calculatesthe proportion lines.
 9. The image processing device according to claim1, wherein the image analyzer calculates at least three or more of theproportion lines, and wherein the image generator changes, in a case ofhaving received an instruction for idealization of facial balance of thefacial image, the positions of the proportion lines to where thedistances between the proportion lines are at a ratio decidedbeforehand, and generates a facial image where at least one of the formand position of a facial part in the facial image has been changed basedon the positions of the proportion lines after changing.
 10. The imageprocessing device according to claim 1, wherein the image generatoroutputs the simulation image to a predetermined display device.
 11. Animage processing method, comprising: inputting a facial image from apredetermined device; calculating one of a facial shape, proportionlines that are lines drawn on the face to analyze the balance of theface, and blocking lines that divide the face into multiple regionsfollowing the structure of the face according to lightness and darknessof shadows arising due to light striking the face, based on facialfeature points extracted from the facial image; determining asuperimposing region of a makeup parts image in the facial image, basedon one of the facial shape, the proportion lines, and the blockinglines; generating a simulation image where the makeup parts image hasbeen superimposed on the superimposing region; calculating an overspreadregion where the facial shape spreads out from a standard facial shapethat has been set beforehand, and an insufficient region where thefacial shape does not fill out the standard facial shape; determining aregion of the facial image corresponding to the overspreading region tobe a superimposing region for a first makeup parts image; anddetermining a region of the facial image corresponding to theinsufficient region to be a superimposing region for a second makeupparts image that is different from the first makeup parts image.
 12. Anon-transitory computer-readable recording medium storing an imageprocessing program that causes a computer to execute the operations ofinputting a facial image from a predetermined device, calculating one ofa facial shape, proportion lines that are lines drawn on the face toanalyze the balance of the face, and blocking lines that divide the faceinto multiple regions following the structure of the face according tolightness and darkness of shadows arising due to light striking theface, based on facial feature points extracted from the facial image,determining a superimposing region of a makeup parts image in the facialimage, based on one of the facial shape, the proportion lines, and theblocking lines, and generating a simulation image where the makeup partsimage has been superimposed on the superimposing region, calculating anoverspread region where the facial shape spreads out from a standardfacial shape that has been set beforehand, and an insufficient regionwhere the facial shape does not fill out the standard facial shape,determining a region of the facial image corresponding to theoverspreading region to be a superimposing region for a first makeupparts image, and determining a region of the facial image correspondingto the insufficient region to be a superimposing region for a secondmakeup parts image that is different from the first makeup parts image.